Electronic device with gesture detection system and methods for using the gesture detection system

ABSTRACT

A method in an electronic device, the method includes projecting infrared (“IR”) light from a plurality of light emitting diodes (“LEDs”) disposed proximate to the perimeter of the electronic device, detecting, by a sensor, IR light originating from at least two of the plurality of LEDs reflected from off of a person, and carrying out a function based on the relative strength of the detected IR light from the LEDs.

CROSS-REFERENCE TO RELATED TO APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/834,422, filed Mar. 30, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/055,637, filed Aug. 6, 2018, which is acontinuation of U.S. patent application Ser. No. 15/243,696, filed Aug.22, 2016, which is a continuation of U.S. patent application Ser. No.14/707,991, filed May 8, 2015, which is a continuation of U.S. patentapplication Ser. No. 14/098,884, filed Dec. 6, 2013, which claims thebenefit of U.S. Provisional Application No. 61/876,691, filed Sep. 11,2013, each of which is hereby incorporated by reference herein in itsentirety.

TECHNICAL FIELD

The present disclosure relates to electronic devices having gesturedetection systems and, more particularly, to electronic devices withinfrared light emitting diode gesture detection system and methods forusing the gesture detection system.

BACKGROUND

Mobile devices such as cellular telephones, smart phones, and otherhandheld or portable electronic devices such as personal digitalassistants (“PDAs”), headsets, MP3 players, etc. have become popular andubiquitous. As more and more features have been added to mobile devices,there has been an increasing desire to equip these mobile devices withinput/output mechanisms that accommodate numerous user commands and/orreact to numerous user behaviors. It is of increasing interest thatmobile devices be capable of detecting the presence of, and determiningwith some accuracy the position of, physical objects located outside ofthe mobile devices and, more particularly, the presence and location ofhuman beings (or portions of their bodies, such as their heads or hands)who are using the mobile devices or otherwise are located nearby themobile devices. By virtue of such capabilities, the mobile devices areable to adjust their behavior in a variety of manners that areappropriate given the presence (or absence) and location of the humanbeings and/or other physical objects.

While remote sensing devices such as infrared (or, more accurately,near-infrared) transceivers have been employed in the past in somemobile devices to allow for the detection of the presence and/orlocation of human beings and/or physical objects even when not inphysical contact with the mobile devices, such sensing devices have beenlimited in various respects. In particular, some such near-infraredtransceivers in some such mobile devices are only able to detect thepresence or absence of a human being/physical object within a certaindistance from the given transceiver (e.g., binarily detect that thehuman being/physical object is within a predetermined distance orproximity to the transceiver), but not able to detect thethree-dimensional location of the human being/physical object inthree-dimensional space relative to the transceiver. Also, some suchtransceivers in some such mobile devices are undesirably complicated orrequire large numbers of components in order to operate, which in turnrenders such devices unduly expensive. Many such implementations aremodular in type and thus require user to have prior knowledge of themodule location in order to operate above the module, resulting in aless than optimum experience. Further, those systems are focused on aspecific implementation of gesturing and not designed to be used as partof other systems such as side approach detection, eye tracking, facialillumination, and data transmissions.

DRAWINGS

While the appended claims set forth the features of the presenttechniques with particularity, these techniques may be best understoodfrom the following detailed description taken in conjunction with theaccompanying drawings of which:

FIG. 1 is a perspective view of an electronic device, which is depictedas a mobile device in the drawing.

FIG. 2 shows example components of the electronic device of FIG. 1.

FIG. 3A is a top view of the electronic device of FIG. 1.

FIG. 3B is another top view of the electronic device of FIG. 1.

FIGS. 4-12 show steps that may be carried out according to variousembodiments.

DESCRIPTION

The present disclosure sets forth an electronic device with an infrared(“IR”) light emitting diode (“LED”) gesture detection system and methodsfor using the gesture detection system. Various embodiments of thedistributed IR LED gesture detection system offer users a betterexperience because the gesture detection system is designed to mimicand/or exceed touch screen coverage area but is not in contact with thecover glass. Another benefit for this distributed IR LED detectionsystem is the ability to enhance hand approach detection, where thedevice may detect a user's hand approaching from any side of the device.The enhanced hand approach detection may be used to wake up device. Inaddition, the IR LED gesture detection system is not only capable ofdetecting gestures, but also supports other functionalities. Asdiscussed in more detail below, the IR LED gesture detection system mayalso support hand approach detection, eye movement tracking, facialillumination, and IR proximity functionality to disable a touch screenof the device during phone calls. Accordingly, the IR LED gesturedetection system provides the benefit of additional applications andfunctionalities to a device employing such gesture detection system.

In an embodiment, an electronic device includes LEDs disposed proximateto the perimeter of the device and a sensor disposed in the device. TheLEDs project IR light. The sensor detects IR light originating from atleast one of the LEDs reflected from off of a person. Based on therelative strength of the detected IR light, the device carries out afunction based on the detected IR light from two selected LEDs. Thedevice may detect a gesture using at least two of the LEDs (i.e., theactive LEDs).

The device may identify, based on the relative strength of the detectedIR light, at least one LED having much weaker detected IR light. Thedevice may ignore that LED or LEDs. In another embodiment, the devicemay ignore at least one LED whose projected IR light is not detected bythe sensor (e.g., due to blockage by a person's hand).

In another embodiment, if the detected IR light originating from oneLEDs becomes weaker than the detected IR light of at least one of theremaining LEDs, the device may reselect at least two other LEDs. Afterreselection, the device may carry out a function based on detected IRlight from the two reselected LEDs. The two reselected LEDs may includethe other of the two selected LEDs having the stronger detected IRlight. This embodiment is directed to using subgroups of the pluralityof LEDs, and dynamically switching between the LED subgroups, to detectgestures.

In a further embodiment, the device illuminates the person's face withone or more LEDs for more diffused type or broad illumination (i.e., afacial illumination function). The LED (or LEDs) projects IR light ontothe person's face, and the sensor receives IR light reflected off of theperson's face. Based on a characteristic of the received IR light, thedevice determines the uniformity of the facial illumination. Based onthe determined uniformity, the device may increase or decrease theprojection of the LED (or LEDs).

The characteristic of the received IR light may include the relativestrength of the received IR light. Based on the relative strength of thereceived IR light, the device may further determine a difference betweenthe received IR light from the LED having the stronger reflected IRlight and the LED having the weaker reflected IR light. Based on thedetermined difference, the device may increase the IR light projectionof the LED having the weaker reflected IR light. This embodiment isdirected to an IR transceiver (LED and sensor) closed loop system, whichmay be used to enhance the facial illumination function. The IRtransceiver closed loop system adaptively adjusts the LEDs' IR lightprojections and coverage based on the received IR light signal toprovide a more uniform illumination of a person's face.

In yet another embodiment, an electronic device includes LEDs disposedproximate to the perimeter of the device and a sensor disposed in thedevice. The LEDs project IR light. The sensor detects IR lightoriginating from at least one of the LEDs reflected from off of aperson. Based on the detected IR light, the device detects the presenceof the person. If the presence of the person is detected, the devicecarries out a function.

In an embodiment, the LEDs sequentially project IR light. If thepresence of the person is detected, the device may turn on a screen,wake up the electronic device, display the time, initiate anotification, change a user interface, or reduce power consumption. Thesensor detects IR light originating from at least two of the LEDsreflected from off of the person and use the reflected light to detect agesture.

In another embodiment, the LEDs simultaneously project IR light todetect a person from a longer range.

In an embodiment, the electronic device includes a first group of LEDsthat are proximate to the perimeter of the device, a second group ofLEDs proximate to the perimeter of the device, and a sensor in thedevice. The first group of LEDs and the second group of LEDs project IRlight. The sensor then detects IR light reflected off of a person. Thedevice then generates signals based on the detected IR light. If thedetected IR light originates from any of the LEDs of the first group,the device processes the signals as a single, first LED. If the detectedIR light originates from any of the LEDs of the second group, the deviceprocesses the signals as a single, second LED. Based on whether thesignals are processed as the first LED or the second LED, the devicedetermines whether the person is making a first gesture or a secondgesture.

In another embodiment, an electronic device first detects itsorientation. Based on the detected orientation, the device activates twoof the LEDs or a subgroup of the LEDs (the subgroup may have two or moreLEDs), which are disposed proximate to the perimeter of the device. Asensor disposed in the device detects reflected IR light originatingfrom at least one of the two activated LEDs. Based on the detected IRlight, the device carries out a function.

If the orientation of the device is detected to be a portrait mode, thedevice activates a first set of two LEDs. If the orientation of thedevice is detected to be a landscape mode, the device then activates asecond set of two LED.

If the orientation of the electronic device is detected to be a portraitmode, both LEDs in the first set of two LEDs may simultaneously projectIR light. If the orientation of the electronic device is detected to bea landscape mode, both LEDs in the second set of two LEDs maysimultaneously project IR light.

In still another embodiment, an electronic device includes a first LEDdisposed proximate to the perimeter of the device, a second LED disposedproximate to the perimeter of the device, and a sensor disposed closerto the first LED than the second LED. The first LED projects IR light,and the second LED projects IR light. The sensor detects reflected IRlight from off of a person from both LEDs. The first LED (disposedcloser to the sensor) allows detection down to glass (e.g., detect aperson touching the glass). The second LED (disposed further to thesensor) is expected to cover a range before the person touches theglass. Based on the detected IR light, the device carries out a votingfunction and determines whether to deactivate a touch screen of thedevice based on the results of the voting function.

To determine whether to deactivate the touch screen, if the detected IRlight originates from the second LED, the device may detect that theperson is approaching the device. If the detected IR light originatesfrom the first LED, the device may deactivate the touch screen. In anembodiment, the second LED may have a longer range than the first LED.

In a further embodiment, an electronic device includes a first LEDdisposed proximate to the perimeter of the electronic device, a secondLED disposed proximate to the perimeter of the device, and a camera. Thefirst LED projects IR light onto a person's eye, and the second LEDprojects IR light onto the person's eye. The camera captures IR lightspots from the first LED and the second LED on the person's eye. Thedevice then tracks the captured IR light spots to detect movement of theeye.

The device may include at least four LEDs disposed proximate to theperimeter of the device. Then device may determine which group of twoLEDs is the most appropriate for tracking movement of the eye. Thedevice may identify an LED that is weaker than the remaining LEDs. Thedevice may then energize the identified LED via the IR transceiver (LEDand sensor) closed loop system described above with respect to thefacial illumination function.

The device may further determine a characteristic of the captured IRlight spots. The characteristic of the captured IR light spot mayinclude the strength of the captured IR light spots to determine thelocation of the person's head relative to the device. The characteristicmay also include the angle at which the captured IR light spots arereceived. The characteristic may further include the direction fromwhich the captured IR light spots are received.

FIG. 1 is a perspective view of an electronic device 100 (also referredto as “device 100” or “device”) according to an embodiment, which isdepicted as a mobile device in the drawing. The electronic device 100includes a housing 110, LEDs disposed proximate to the perimeter of thehousing, and a sensor 130. The housing 110 includes a cover glass and anopening 112 formed therein, and a speaker is disposed in the housingbelow the opening 112. In the embodiment shown, the LEDs include a firstLED 120, a second LED 122, a third LED 124, and a fourth LED 126. Asshown in FIG. 1, the first LED 120, the second LED 122, the third LED124, and the fourth LED 126 are disposed proximate to the four cornersof the housing 110, respectively.

Although FIG. 1 illustrates the housing 110 as having a substantiallyrectangular shape including sides and rounded corners, in otherembodiments, the housing 110 may have another shape without sides and/orcorners. Furthermore, while FIG. 1 shows four LEDs located proximate tothe corners of the electronic device 100, in other embodiments, the LEDsmay be disposed proximate to other areas of the perimeter of the device100, instead of the corners of the device 100.

Each of the LEDs projects or emits IR light having a wavelength thattypically ranges from about 850 nanometers to about 1 micrometer forinfrared. The LEDs could project light having a wavelength that rangeslower into the visible region as well, e.g., down toward 400 nm range.The sensor 130, which is an IR light sensor or receiver, detects IRlight from the LEDs.

In one embodiment, the LEDs are of the same color. The LEDs may projector emit IR light sequentially, or the LEDs may project or emit lightfollowing a pulse or wave-shaped pattern. In another embodiment, theLEDs are of different colors, and may project or emit lightsimultaneously.

In the embodiment shown in FIG. 1, The LEDs are equipped with microfilmbeam bending lenses to direct the IR light beams projected by the LEDsin certain directions. The sensor 130 is disposed proximate to theopening 112 (i.e., proximate to the speaker). The sensor 130 is also bedisposed closer to the first LED 120 than to the remaining three LEDs toperform an IR proximity function, which will be described in detailbelow.

FIG. 2 shows internal components of the device 100 of FIG. 1, inaccordance with an embodiment of the disclosure. As shown in FIG. 2, theinternal components 200 include one or more wireless transceivers 202, aprocessor 204 (e.g., a microprocessor, microcomputer,application-specific integrated circuit, etc.), a memory portion 206,one or more output devices 208, and one or more input devices 210. Theinternal components 200 can further include a component interface 212 toprovide a direct connection to auxiliary components or accessories foradditional or enhanced functionality. The internal components 200 mayalso include a power supply 214, such as a battery, for providing powerto the other internal components while enabling the mobile device to beportable. Further, the internal components 200 additionally include oneor more sensors 228. All of the internal components 200 can be coupledto one another, and in communication with one another, by way of one ormore internal communication links 232 (e.g., an internal bus).

Further, in the embodiment of FIG. 2, the wireless transceivers 202particularly include a cellular transceiver 203 and a Wi-Fi transceiver205. More particularly, the cellular transceiver 203 is configured toconduct cellular communications, such as 3G, 4G, 4G-LTE, vis-à-vis celltowers (not shown), albeit in other embodiments, the cellulartransceiver 203 can be configured to utilize any of a variety of othercellular-based communication technologies such as analog communications(using AMPS), digital communications (using CDMA, TDMA, GSM, iDEN, GPRS,EDGE, etc.), and/or next generation communications (using UMTS, WCDMA,LTE, IEEE 802.16, etc.) or variants thereof.

By contrast, the Wi-Fi transceiver 205 is a wireless local area network(WLAN) transceiver 205 configured to conduct Wi-Fi communications inaccordance with the IEEE 802.11 (a, b, g, or n) standard with accesspoints. In other embodiments, the Wi-Fi transceiver 205 can instead (oradditionally) conduct other types of communications commonly understoodas being encompassed within Wi-Fi communications, such as some types ofpeer-to-peer (e.g., Wi-Fi Peer-to-Peer) communications. Further, inother embodiments, the Wi-Fi transceiver 205 can be replaced orsupplemented with one or more other wireless transceivers configured fornon-cellular wireless communications including, for example, wirelesstransceivers employing ad hoc communication technologies such as HomeRF(radio frequency), Home Node B (3G femtocell), Bluetooth and/or otherwireless communication technologies such as infrared technology.

Although in the embodiment shown in FIG. 1, the device 100 has two ofthe wireless transceivers 202 (that is, the transceivers 203 and 205),the present disclosure is intended to encompass numerous embodiments inwhich any arbitrary number of wireless transceivers employing anyarbitrary number of communication technologies are present. By virtue ofthe use of the wireless transceivers 202, the device 100 is capable ofcommunicating with any of a variety of other devices or systems (notshown) including, for example, other mobile devices, web servers, celltowers, access points, other remote devices, etc. Depending upon theembodiment or circumstance, wireless communication between the device100 and any arbitrary number of other devices or systems can beachieved.

Operation of the wireless transceivers 202 in conjunction with others ofthe internal components 200 of the device 100 can take a variety offorms. For example, operation of the wireless transceivers 202 canproceed in a manner in which, upon reception of wireless signals, theinternal components 200 detect communication signals and thetransceivers 202 demodulate the communication signals to recoverincoming information, such as voice and/or data, transmitted by thewireless signals. After receiving the incoming information from thetransceivers 202, the processor 204 formats the incoming information forthe one or more output devices 208. Likewise, for transmission ofwireless signals, the processor 204 formats outgoing information, whichcan but need not be activated by the input devices 210, and conveys theoutgoing information to one or more of the wireless transceivers 202 formodulation so as to provide modulated communication signals to betransmitted.

Depending upon the embodiment, the input and output devices 208, 210 ofthe internal components 200 can include a variety of visual, audioand/or mechanical outputs. For example, the output device(s) 208 caninclude one or more visual output devices 216 such as a liquid crystaldisplay and/or light emitting diode indicator, one or more audio outputdevices 218 such as a speaker, alarm, and/or buzzer, and/or one or moremechanical output devices 220 such as a vibrating mechanism. The visualoutput devices 216 among other things can also include a video screen.Likewise, by example, the input device(s) 210 can include one or morevisual input devices 222 such as an optical sensor (for example, acamera lens and photosensor), one or more audio input devices 224 (orfurther for example a microphone of a Bluetooth headset), and/or one ormore mechanical input devices 226 such as a flip sensor, keyboard,keypad, selection button, navigation cluster, touch pad, capacitivesensor, motion sensor, and/or switch. Operations that can actuate one ormore of the input devices 210 can include not only the physicalpressing/actuation of buttons or other actuators, but can also include,for example, opening the mobile device, unlocking the device, moving thedevice to actuate a motion, moving the device to actuate a locationpositioning system, and operating the device.

As mentioned above, the internal components 200 also can include one ormore of various types of sensors 228 as well as a sensor hub to manageone or more functions of the sensors. The sensors 228 may include, forexample, proximity sensors (e.g., a light detecting sensor, anultrasound transceiver or an infrared transceiver), touch sensors,altitude sensors, and one or more location circuits/components that caninclude, for example, a Global Positioning System (GPS) receiver, atriangulation receiver, an accelerometer, a tilt sensor, a gyroscope, orany other information collecting device that can identify a currentlocation or user-device interface (carry mode) of the device 100.Although the sensors 228 for the purposes of FIG. 2 are considered to bedistinct from the input devices 210, in other embodiments it is possiblethat one or more of the input devices can also be considered toconstitute one or more of the sensors (and vice-versa). Additionally,although in the present embodiment the input devices 210 are shown to bedistinct from the output devices 208, it should be recognized that insome embodiments one or more devices serve both as input device(s) andoutput device(s). In particular, if the device 100 includes a touchscreen display, the touch screen display can be considered to constituteboth a visual output device and a mechanical input device.

The memory portion 206 of the internal components 200 can encompass oneor more memory devices of any of a variety of forms (e.g., read-onlymemory, random access memory, static random access memory, dynamicrandom access memory, etc.), and can be used by the processor 204 tostore and retrieve data. In some embodiments, the memory portion 206 canbe integrated with the processor 204 in a single device (e.g., aprocessing device including memory or processor-in-memory (PIM)), albeitsuch a single device will still typically have distinctportions/sections that perform the different processing and memoryfunctions and that can be considered separate devices. In some alternateembodiments, the memory portion 206 of the device 100 can besupplemented or replaced by other memory portion(s) located elsewhereapart from the mobile device and, in such embodiments, the mobile devicecan be in communication with or access such other memory device(s) byway of any of various communications techniques, for example, wirelesscommunications afforded by the wireless transceivers 202, or connectionsvia the component interface 212.

The data that is stored by the memory portion 206 can include, but neednot be limited to, operating systems, programs (applications), modules,and informational data. Each operating system includes executable codethat controls basic functions of the device 100, such as interactionamong the various components included among the internal components 200,communication with external devices via the wireless transceivers 202and/or the component interface 212, and storage and retrieval ofprograms and data, to and from the memory portion 206. As for programs,each program includes executable code that utilizes an operating systemto provide more specific functionality, such as file system service andhandling of protected and unprotected data stored in the memory portion206. Such programs can include, among other things, programming forenabling the device 100 to perform a process such as the process forgesture recognition and discussed further below. Finally, with respectto informational data, this is non-executable code or information thatcan be referenced and/or manipulated by an operating system or programfor performing functions of the device 100.

Referring to FIG. 1, the first LED 120, the second LED 122, the thirdLED 124, the fourth LED 126, and the sensor 130 together form an IRgesture detection system. In an embodiment, each of the plurality ofLEDs projects IR light, and the sensor 130 detects IR light reflected bya person or an object. Based on the reflected IR light, the device 100selects at least two of the LEDs to detect gesture. In one embodiment,the device 100 selects two LEDs (e.g., the first LED 120 and the secondLED 122) to detect a two-dimensional (“2D”) gesture. In anotherembodiment, the device 100 selects three LEDs (e.g., the first LED 120,the second LED 122, and the third LED 124) to detect a three-dimensional(“3D”) gesture.

In one embodiment, the device 100 requires no more than three LEDs todetect gestures. Thus, if the reflected IR light from one of theplurality of LEDs is weaker than the remaining LEDs (e.g., the first LED120 is blocked by a person's hand), the device 100 may select two orthree of the remaining LEDs for gesture detection (e.g., the second LED122, the third LED 124, and the fourth LED 126). In other words, thedevice 100 may ignore the LED having the weakest reflected IR light.Based on the reflected IR light received by the sensor 130, the device100 may thus adaptively select the LEDs to be used for gesture detectionas the person uses or grabs the device 100.

In yet another embodiment, the device 100 may detect a personapproaching the device 100, e.g., the presence of the person near thedevice 100. In this embodiment, the plurality of LEDS project IR light.The sensor 130 detects IR light originating from at least one of theplurality of LEDs (e.g., the first LED 120) reflected from off of theperson. Based on the detected IR light from LED or LEDs, the device 100detects whether the person is present and where the person is locatedwith respect to the device 100 (e.g., direction and distance of theperson with respect to the device 100). If the presence of the person isdetected, the device 100 may carry out a function, e.g., turning on ascreen, waking up the electronic device, displaying the time, initiatinga notification, changing a user interface, and reducing powerconsumption.

In this embodiment, the plurality of LEDs project IR light pulsessequentially or serially. By cycling through the plurality of LEDs, thesensor 130 may detect any disturbance (e.g., a person's hand reflectingIR light projected by at least one of the LEDs) to indicate the person'shand approach. In response to the person's hand approach, the device 100may turn on its display, initiate notification, or alert the person ofmessages.

In another embodiment, the plurality of LEDs or subset of them mayproject IR light simultaneously. This configuration increases the rangeor coverage of the plurality of LEDs and thus enhances hand approachdetection.

In still another embodiment, the device 100 may cycle the plurality ofLEDs for person or object detection, perform gesture detection, andenhanced hand approach detection. First, the plurality of LEDs projectsIR light pulses serially, and the sensor 130 detects reflected IR lightfrom any LED to detect a person or object. If the sensor 130 detectsreflected IR light from only one LEDs or any combination of LEDs, thesensor 130 detects the presence of the person or object. Then, theplurality of LEDs project IR light pulses serially to detect gestures,and the sensor 130 detects reflected IR light from at least two of theLEDs. Next, some of the plurality of LEDs transmits IR light pulsessimultaneously for enhanced hand approach detection. The LEDs are thenturned off for a preset period of time (e.g., 20 milliseconds). Thecycle may then be repeated. In this embodiment, the duration of theprojected IR pulses is about 10 microseconds. In other embodiments,however, the duration of the projected IR pulses may vary.

FIG. 3A is a top view of the electronic device of FIG. 1. In anembodiment, the device 100 includes an accelerometer to detect anorientation of the device 100. Based on the orientation of the device100, i.e., portrait mode or landscape mode, the plurality of LEDs may begrouped into two or more independent gesture detection systems.

As shown in FIG. 3A, the device 100 is in portrait mode. If the detectedorientation of the device 100 is portrait mode, the first LED 120 andthe second LED 122 may form into a first group 310, and the third LED124 and the fourth LED 126 may form into a second group 320. Each of thefirst group 310 and the second group 320 functions as an independent 2Dgesture detection system. If a person conducts a 2D gesture (e.g.,swiping motion from left to right without moving toward or away from thedevice 100) near the top portion of the device 100, the first group 310will detect the gesture. If the person conducts a 2D gesture near thebottom portion of the device 100, the second group 320 will detect thegesture.

If the detected orientation of the device is landscape mode (e.g., thedevice 100 shown in FIG. 3A rotated 90° in the clockwise orcounterclockwise direction), the second LED 122 and the fourth LED 126may form into a third group 330, and the first LED 120 and the third LED124 may form into a fourth group 340. Each of the third group 330 andthe fourth group 340 functions as an independent 2D gesture detectionsystem. If a person conducts a 2D gesture near the top portion of thedevice 100 in landscape mode (e.g., the right portion of the device 100as shown in FIG. 3A), the third group 330 will detect the gesture. Ifthe person conducts a 2D gesture near the bottom portion of the device100 in landscape mode (e.g., the left portion of the device 100 as shownin FIG. 3A), the fourth group 340 will detect the gesture.

FIG. 3B is another top view of the electronic device of FIG. 1,according to an embodiment, in which the detected orientation of thedevice 100 is landscape mode. The device 100 includes two LED groups;the first group 310 includes the first LED 120 and the second LED 122,and the second group 320 includes the third LED 124 and the fourth LED126. Unlike the system illustrated in FIG. 3A, the first group 310 andthe second group 320 do not function as two independent 2D gesturedetection systems. Instead, the two LEDs in each group function as asingle LED. In other words, the first LED 120 and the second LED 122 ofthe first group 310 functions as a single LED; the third LED 124 and thefourth LED 126 of the second group 320 function as another signal LED.Together, the first group 310 and the second group 320 constitute asingle 2D gesture detection system. For example, when a person conductsa 2D gesture (e.g., swiping motion from left to right), the person'shand must reflect IR light projected by each of the first group 310 andthe second group 320 in order to be detected by the sensor 130.

In an embodiment, although the LEDs projects IR light sequentially, thesensor 130 interprets the signals from the LEDs of the first grouptogether and interprets the signals from the LEDs of the second grouptogether. In another embodiment, the LEDs in each group project IR lightsimultaneously (e.g., the first LED 120 and the second LED 120 projectsimultaneously, and the third LED 124 and the fourth LED 126 projectsimultaneously). In the simultaneously projection embodiment, IR lightprojected by the LEDs in each group is not distinguished, and light fromall of the LEDs is treated as the same light, having double the opticaloutput power.

While FIGS. 3A and 3B illustrate two LEDs in each group, in otherembodiments, each group of LEDs includes three LEDs to detect 3Dgestures. Furthermore, although FIG. 3B only shows the first group 310and the second group 320, in other embodiments, the third group 330 andthe fourth group 340 (as shown in FIG. 3B) constitute a single 2Dgesture detection system.

FIGS. 4-12 show steps that are carried out by the electronic device 100according to various embodiments.

Referring to FIG. 4, a procedure 400 for gesture detection is carriedout by the electronic device 100. At step 402, the electronic device 100projects IR light from the LEDs (e.g., the first LED 120, the second LED122, the third LED 124, and the fourth LED 126). At step 404, the sensor130 of the device 100 detects IR light originating from at least two ofthe LEDs reflected from off of a person. At step 406, the device 100determines the relative strength, from among the LEDs, of the detectedIR light. Then, at step 408, the device 100 carries out a function basedon the determined relative strength of the detected IR light.

In an embodiment, at step 408, the device 100 may detect a gesture usingthe light from all of the LEDs. In another embodiment, at step 408, thedevice 100 may ignore the LEDs whose light have not been detected by thesensor 130 (e.g., the LEDs may be blocked by a person's hand). Using therelative strength of the detected IR light, the device 100 may identifythe LEDs having the weaker reflected IR light. The weakly received lightinformation may be helpful, as it may serve as an additional data pointwhen the device 100 is performing various functions (e.g., identifyingthe LEDs having the weaker reflected IR light may allow the device 100to further determine a location of an object or person with respect tothe device). In other embodiments, the device 100 may ignore the LEDshaving the weaker reflected IR light.

In yet another embodiment, the device 100 selects three LEDs instead oftwo LEDs for 3D gesture detection. Then, using the three selected LEDs,the device 100 may detect a 3D gesture.

Referring to FIG. 5, a procedure 500 for reselecting IR LEDs is carriedout by the electronic device 100. At step 502, the electronic device 100projects IR light from the LEDs. At step 504, the sensor 130 of thedevice 100 detects IR light reflected from off of a person. At step 506,the device 100 selects, based on relative strength of the detected IRlight, at least two of the LEDs having the stronger reflected IR light.Then, at step 508, the device 100 carries out a function based on thedetected IR light from the selected LEDs.

Next, at step 510, the device 100 determines whether the detected IRlight originating from one of the selected LEDs has become weaker thanthe detected IR light of at least one of the remaining LEDs (e.g.,whether one of the two selected LEDs is blocked by the person's handwhile the person is handling the device 100). If the device 100determines that the detected IR light originating from one of theselected LEDs has not become weaker than the remaining LEDs (NO of step510), the procedure returns to step 508 where the device 100 carries outa function based on the detected IR light from the selected LEDs.

If the device 100 determines that the detected IR light originating fromone selected LEDs has become weaker than the remaining LEDs (YES of step510), the device 100 reselects at least two other LEDs at step 512.After reselection, at step 514, the device 100 carries out a functionbased on detected IR light from the reselected LEDs. The reselected LEDsmay include LEDs from the originally selected LEDs having the strongerdetected IR light. The device 100 may repeat steps 510 to 514 in orderto continually select the LEDs having the stronger reflected light tocarry out the function.

Referring to FIG. 6, a procedure 600 is carried out by the electronicdevice 100. At step 602, the electronic device 100 projects IR lightfrom the LEDs. At step 604, the sensor 130 of the device 100 detects IRlight originating from at least one of the LEDs reflected from off of aperson. At step 606, the device 100 selects, based on the detected IRlight, two LEDs having the stronger reflected IR light. At step 608, thedevice 100 may ignore the LEDs having no reflected light.

At step 610, the device 100 may determine, based on the detected IRlight, a difference between the detected IR light from the two selectedLEDs having the stronger reflected IR light and the remaining LEDshaving the weaker reflected IR light. Based on the determineddifference, at step 612, the device may increase the IR light projectionof the remaining LEDs having the weaker reflected IR light.

Referring to FIG. 7, a procedure 700 is carried out by the electronicdevice 100. Procedure 700 is directed to a facial and/or objectillumination function using the IR LED system. Image recognitionfunction is highly impacted by facial and/or object illumination.Without adequate illumination, image recognition failures are prevalent.Following the steps set forth in procedure 700, the electronic device100 may create a uniform broad illumination of a person's face.Furthermore, when combined with procedure 500 (i.e., repeated selectionof LEDs having the stronger reflected IR light), the device 100 may befurther optimized to drive LEDs to compensate for poor user-to-deviceorientations.

In more detail, at step 702, the electronic device 100 projects IR lightthe LEDs. At step 704, the sensor 130 of the device 100 detects IR lightreflected from off of a person. At step 706, the device 100 selects,based on the relative strength of the detected IR light, at least two ofthe LEDs (e.g., LEDs having the stronger reflected IR light). In anotherembodiment, the device 100 may select one LED, e.g., the LED with thestrongest reflected IR light, and use the selected LED to illuminate theperson's face.

Next, at step 708, the device 100 projects IR light from the selectedLEDs onto the person's face. The sensor 130 of the device receives IRlight reflected off of the person's face at step 710. Based on acharacteristic of the received IR light, the device 100 determines theuniformity of the facial illumination at step 712. Then at step 714,based on the determined uniformity, the device 100 increases ordecreases the projection of one or both of the two selected LEDs.

In an embodiment, the characteristic of the received IR light mayinclude the relative strength of the received IR light from the LEDs.Based on the relative strength of the received IR light, the device 100may further determine a difference between the received IR light from atleast one LED having the stronger reflected IR light and at least oneLED having the weaker reflected IR light. Based on the determineddifference, the device 100 may increase the IR light projection of theLED or LEDs having the weaker reflected IR light. This embodiment isdirected to an IR transceiver (LED and sensor) closed loop system, whichmay be used to enhance the facial illumination function as shown inprocedure 700. The IR transceiver closed loop system adaptively adjuststhe LEDs' IR light projections and coverage based on the received IRlight signal to provide a more uniform illumination of the person'sface.

Referring to FIG. 8, a procedure 800 is carried out by the electronicdevice 100. Procedure 800 is directed to detecting gestures using IR LEDgroups. More specifically, in procedure 800, each group of LEDsfunctions as a single LED.

At step 802, the electronic device 100 projects IR light from a firstgroup of LEDs. At step 804, the device 100 projects IR light from asecond group of LEDs. The sensor 130 of the device 100 then detects IRlight reflected off of a person at step 806. The device 100 thengenerates signals based on the detected IR light at step 808.

At step 810, if the device 100 determines that the detected IR lightoriginates from any of the LEDs of the first group, the device 100processes the signals as a single, first LED. At step 812, if the device100 determines that the detected IR light originates from any of theLEDs of the second group, the device 100 processes the signals as asingle, second LED. At step 814, based on whether the signals areprocessed as the first LED or the second LED, the device 100 determineswhether the person is making a first gesture or a second gesture.

Referring to FIG. 9, a procedure 900 is carried out by the electronicdevice 100. Procedure 900 is also directed to detecting gestures usingIR LED groups. Unlike procedure 800, however, the LED groups inprocedure 900 functions as independent gesture detection systems.

At step 902, the electronic device 100 detects its orientation. At step904, if the orientation of the device 100 is detected to be a portraitmode, the device 100 activates a first set of two LEDs. At step 906, ifthe orientation of the device is detected to be a landscape mode, thedevice 100 then activates a second set of two LEDs.

After activation, at step 908, the device projects light from either thefirst set of LEDs or the second set of LEDs. At step 910, the sensor 130of the device 100 detects reflected IR light originating from at leastone of the two activated LEDs reflected off of a person. Based on thedetected IR light, at step 912, the device 100 carries out a functionbased on the detected reflected IR light from the activated set of LEDs.

In another embodiment, after activation and based on the detectedorientation, the device may treat each set of LEDs as a single LED. Forexample, if the orientation of the electronic device 100 is detected tobe a portrait mode, both LEDs in the first set of two LEDs maysimultaneously project IR light. If the orientation of the electronicdevice 100 is detected to be a landscape mode, both LEDs in the secondset of two LEDs may simultaneously project IR light.

Referring to FIG. 10, a procedure 1000 is carried out by the electronicdevice 100. Procedure 1000 is directed to an IR proximity function. TheIR proximity systems of conventional devices have a short range (aboutan inch) to eliminate the possibility of unintended false detection whenan object or person comes near the devices. The short range may beproblematic as a user sometimes moves the device toward his or her facein an odd orientation resulting in unintended actuation of the touchscreen (proximity detection failure) before his or her face is detected.

In an embodiment, to perform the IR proximity function, two LEDs fromthe IR LED gesture detection system are used. The two LEDs shouldinclude the LED that is disposed closest to the sensor. For example, inthe device 100 shown in FIG. 1, the first LED 120, which is disposedclosest to the sensor 130, should be utilized for performing the IRproximity function. The other LED may be the second LED 122 or thefourth LED 126, which are disposed closer to the first LED 120 than thethird LED 124.

The spacing between the first LED 120 and the sensor 130 may vary basedon the distance between the cover glass of the housing 110 and the firstLED 120, the distance between the cover glass of the housing 110 and thesensor 130, the projection area of the first LED 120, and the receptionarea of the sensor 130. To detect the presence of an object or a personat a close range (e.g., down to touching the cover glass such as thecase when the person is conducting a phone call), the projection area ofthe first LED 120 and the reception area of the sensor 130 shouldintersect at or below the cover glass of the housing 110. Accordingly,if the first LED 120 has a large projection area or if the sensor 130has a large reception area, the first LED 120 or the sensor 130 may bedisposed closer to the cover glass of the housing 110.

In one embodiment, the sensor 130 of the IR LED gesture detection systemis located in an area where the sensor 130 may detect a person's facetouching the device 100 (e.g., near the speaker opening 112 of the topsurface of the housing 110). If the sensor 130 is disposed at the bottomor sides of the device 100, the IR LED gesture detection system willstill detect IR proximity, but the IR proximity detection function maybe falsely triggered by the user's hand carrying device 100 or dialingthe device 100, etc.

In addition, using two IR LEDs provides a voting ability of facepresence detection that a single LED cannot. For instance, during aphone call, the first LED 120 and the second LED 122 may be used forperforming the IR proximity function. The first LED 120 and the secondLED 122 are cycled in time (e.g., TDMA). If reflected IR light fromeither of the first LED 120 and the second LED 122 is detected by thesensor 130, the device 100 determines that a face is present and maydeactivate the touch screen. However, because the second LED 122 isfarther away from the sensor 130 than the first LED 120, the second LED124 is used to detect a person's approach before the person touches thecover glass of the housing 110 and supplements the first LED 120, whichcan detect both approach and touch (intercept point design). In otherembodiments, the first and second LEDs may be disposed right next tosensor (e.g., on either side of the sensor), and both LEDs may detectboth approach and touch.

Referring to FIG. 10, at step 1002, the electronic device 100 projectsIR light from a first LED (i.e., the first LED 120). At step 1004, thedevice 100 projects IR light from a second LED (e.g., the second LED 122or the fourth LED 126). At step 1006, the sensor 130 of the device 100detects reflected IR light from off of a person. In the presentembodiment, the sensor 130 is disposed closer to the first LED than thesecond LED (e.g., the sensor 130 is disposed closest to the first LED120).

The device 100 may use the two LEDs in procedure 1000 to carry out avoting function based on whether the detected IR light originates fromthe first LED or the second LED. Based on the result of the votingfunction, the device 100 may determine whether to deactivate the touchscreen. To carry out the voting function, one of the two LEDs may have alonger projection range than the other of the two selected LEDs, or oneof the two LEDs may be disposed closer to the sensor 130 than the other.In such an embodiment, the device 100 will only deactivate the touchscreen if the detected IR light originates from the first LED, and willcarry out another function if the detected IR light originates from thesecond LED. For instance, at step 1008, if the detected IR lightoriginates from the first LED, the device 100 deactivates a touch screenof the device 100. At step 1010, if the device 100 determines that thedetected IR light originates from the second LED (which is disposedfarther from the sensor 130 than the first LED), the device 100 maydetects the presence of the person and may perform a notificationfunction indicating that a person is present or approaching the device100.

Referring to FIG. 11, a procedure 1100 is carried out by the electronicdevice 100. Procedure 1100 is directed to an eye movement trackingfunction. During eye movement tracking, two LEDs from the IR LED gesturesystem are selected to project IR light onto a person's eye pupil.Projected IR light spots will appear on the eye. The device 100 thencaptures the IR light spots on the eye using a front facing camera ofthe device. For effective eye movement tracking, multiple LEDs areneeded for depth detection. Furthermore, these LEDs should be disposedapart in order to appear as separate sources of IR light (e.g., about2.5 inches of separation between the LEDs may be preferred).

Referring to FIG. 11, at step 1102, the electronic device 100 projectsIR light from a first LED onto a person's eye. At step 1104, the device100 projects IR light from a second LED. Then at step 1106, the cameraof the device 100 captures IR light spots on the person's eye from thefirst LED and the second LED. The device 100 tracks the IR light spotsto detect movement of the eye in step 1108.

The device 100 may further determine a characteristic of the captured IRlight spots. The characteristic of the captured IR light spots mayinclude the strength of the captured IR light spots. The characteristicmay also include the angle at which the captured IR light spots isreceived. The characteristic may further include the direction fromwhich the captured IR light spots is received. Using the determinedcharacteristic, the IR LED detection system may dynamically adapt to anychanges in the orientation or direction of the person's head or eyes.

In an embodiment, before performing procedure 1100, the device 100 maydetermine which group of two LEDs of the plurality of LEDs is the mostappropriate for tracking movement of the eye. In the IR LED gesturedetection system described above, the device 100 may select andadaptively switch to the most suitable LEDs for tracking movement of theeye based on a user's looking direction. The user's looking directionrelative to device 100 may be using the procedure 400 directed to IR LEDselection. Thus, this IR LED selection for eye movement tracking issimilar to the selection and reselection of LEDs described with respectto FIGS. 4 and 5. In this case, a camera of the device 100 may be usedinstead of the sensor 130.

Furthermore, as discussed in with respect to FIG. 5, the reflected IRlight from the selected LEDs is continuously monitored. This proceduremay also be adapted to the eye movement tracking function. When thereflected IR light from a selected LED is weak, the device 100 mayenergize the weak LED for better eye movement tracking based on the faceor object location relative to the device 100. In other words, thedevice 100 may identify an LED of the plurality of LEDs that is weakerthan the remaining LEDs, and the device 100 may then energize theidentified LED. Accordingly, the IR LED gesture detection system helpsto improve eye movement tracking via a continuous feedback loop. The IRLED gesture detection system may also be used to overcome backgroundinterference.

Referring to FIG. 12, a procedure 1200 is carried out by the electronicdevice 100. Procedure 1200 is directed to cycling the plurality of LEDsto detect the approach or presence of a person and detect a gesture.First the device 100 detects whether the person is present orapproaching the device 100. At step 1202, the LEDs project IR light. Thesensor 130 detects IR light originating from at least one of the LEDsreflected from off of the person at step 1204. Based on the detected IRlight, the device 100 detects the presence of the person at step 1206.Then at step 1208, if the presence of the person is detected (e.g., theperson is approaching the device 100), the device 100 carries out afunction. The device 100 may turn on a screen, wake up the device 100,display the time, initiate a notification, change a user interface,perform power optimization to turn the device 100 on or off, or reducethe power consumption of the device 100.

Once the device 100 has detected the approach or presence of the person,the device 100 detects a gesture of the person, who may be dismissing anotification or may be instructing the device 100 to person anotherfunction. At step 1210, the sensor 130 detects IR light originating fromLEDs reflected from off of the person. Then at step 1212, the device 100detects the gesture of the person using the light of the LEDs.

For both person/object detection and gesture detection, the plurality ofLEDs may project IR light sequentially. In other embodiments, toincrease the detection range or coverage, the plurality of LEDs mayproject IR light simultaneously (e.g., when the device 100 performs theenhanced hand approach detection discussed above).

In addition to the exemplary functions described above, the IR LEDgesture detection system may be adapted for carrying out otherfunctions. For instance, the presence of four LEDs is highly beneficialto reliable IR transmission. When all LEDs projects IR lightsimultaneously, a super transmitter is achieved with longer range andbroader coverage. Thus, when some LEDs are obstructed by user's hand,barcode data can still be transmitted using the unobstructed LED.

It can be seen from the foregoing that an electronic device with an IRLED gesture detection system and methods for using the gesture detectionsystem have been provided. In view of the many possible embodiments towhich the principles of the present discussion may be applied, it shouldbe recognized that the embodiments described herein with respect to thedrawing figures are meant to be illustrative only and should not betaken as limiting the scope of the claims. Therefore, the techniques asdescribed herein contemplate all such embodiments as may come within thescope of the following claims and equivalents thereof.

What is claimed is:
 1. A method in an electronic device, the methodcomprising: detecting, using a sensor, light originating from aplurality of light sources that is reflected from a feature of a user ofthe electronic device; generating a signal based on a plurality ofdetermined characteristics of the detected light; and carrying out afunction on the electronic device based on the signal.
 2. The method ofclaim 1, wherein the carrying out a function on the electronic devicecomprises carrying out a function on the electronic device based onrelative strength of the detected light originating from the pluralityof light sources.
 3. The method of claim 2, further comprisingreselecting at least one light source of the plurality of light sourcesif the detected light originating from at least one of the plurality oflight sources becomes weaker than detected light originating from the atleast one light source.
 4. The method of claim 2, further comprising:determining, based on the relative strength of the detected light, adifference between detected light originating from at least one lightsource of the plurality of light sources having a stronger detectedlight and detected light originating from at least one light source ofthe plurality of light sources having a weaker detected light; andcausing an increased amount of light to be projected from the at leastone light source from which the weaker detected light originated basedon the determined difference.
 5. The method of claim 1, wherein theplurality of determined characteristics includes a strength of thedetected light.
 6. The method of claim 1, wherein the plurality ofdetermined characteristics includes an angle at which the detected lightis detected.
 7. The method of claim 1, wherein the plurality ofdetermined characteristics includes a direction from which the detectedlight is detected.
 8. The method of claim 7, further comprisingreselecting at least one light source of the plurality of light sourcesbased on the at least one signal indicating the direction from which thedetected light is detected.
 9. The method of claim 1, wherein theplurality of light sources is disposed proximate to the perimeter of theelectronic device.
 10. The method of claim 1, wherein the sensor is afront facing camera of the electronic device.
 11. An electronic devicecomprising: a plurality of light sources; a sensor for detecting lightoriginating from one or more of the plurality of light sources; and ahardware processor that is configured to: detect, using a sensor, lightoriginating from a plurality of light sources that is reflected from afeature of a user of the electronic device; generate a signal based on aplurality of determined characteristics of the detected light; and carryout a function on the electronic device based on the signal.
 12. Theelectronic device of claim 11, wherein the carrying out a function onthe electronic device comprises carrying out a function on theelectronic device based on relative strength of the detected lightoriginating from the plurality of light sources.
 13. The electronicdevice of claim 12, wherein the processor is further configured toreselect at least one light source of the plurality of light sources ifthe detected light originating from at least one of the plurality oflight sources becomes weaker than detected light originating from the atleast one light source.
 14. The electronic device of claim 12, whereinthe hardware processor is further configured to: determine, based on therelative strength of the detected light, a difference between detectedlight originating from at least one light source of the plurality oflight sources having a stronger detected light and detected lightoriginating from at least one light source of the plurality of lightsources having a weaker detected light; and cause an increased amount oflight to be projected from the at least one light source from which theweaker detected light originated based on the determined difference. 15.The electronic device of claim 11, wherein the plurality of determinedcharacteristics includes a strength of the detected light.
 16. Theelectronic device of claim 11, wherein the plurality of determinedcharacteristics includes an angle at which the detected light isdetected.
 17. The electronic device of claim 11, wherein the pluralityof determined characteristics includes a direction from which thedetected light is detected.
 18. The electronic device of claim 17,wherein the hardware processor is further configured to reselect atleast one light source of the plurality of light sources based on the atleast one signal indicating the direction from which the detected lightis detected.
 19. The electronic device of claim 11, wherein theplurality of light sources is disposed proximate to the perimeter of theelectronic device.
 20. The electronic device of claim 11, wherein thesensor is a front facing camera of the electronic device.
 21. Anon-transitory computer-readable medium containing computer executableinstructions that, when executed by a processor, cause the processor toperform a method in an electronic device, the method comprising:detecting, using a sensor, light originating from a plurality of lightsources that is reflected from a feature of a user of the electronicdevice; generating a signal based on a plurality of determinedcharacteristics of the detected light; and carrying out a function onthe electronic device based on the signal.